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ABSTRACT ' % . - § * • - 

. The Task Forge on Resource Allocation for Program^ . 
Evaluation was established by Division H of the American Educational 
Research Association at the 1978 Annual Meeting in Toronto. This 
paper^presents' the recommendations of the. Task Force concerning the 
allocation *of resources for program evaluation. Program evaluation is 
needed-to provide information for making program as well as 
policy-making decisions. The results o£ .a "national survey, which 
, examined current vs. perceived-ideal procedures for determining 
pyo^rara evaluation budgets, implied that educational ^valuators need, 
to increase the^r $f forts to communicate evaluation results and . 
encourage educational leaders to use these results*. The Task Force 
recommend^ the adoption of a common definition; of program evaluation; 
that ropiest s for evaluations be accompanied by a statement of the 
use to which the evaluations will be put; .and, that appropriate 
funding levels for conducting evaluations* be set using a sliding 
scale. (Author/PN) >' - & ; 1 » •* 
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RESOURCE AfcLOCATION FOR PROGRAM EVALUATION.: A POSITION * 

BACKGROUND 
« i < 

The Task Force on Resource Allocation for Program Evaluation was established 
by Division H of the American Educational Research Association at the 1978 
Annual Meeting in Toronto. This task force is one. of nine such Division «* 
H cpmmittees .looking into a variety of evaluation questions. . ' , 

This paper presents the recommendations of the Task Forte concerning the - 
question of resource allocation -for (frogram evaluation. The 'first section 
of the paper attempts to establish a ratiprtale or need for the conduct and 
use of program eyal unions % The second section discusses the results of a' 
national sun^y yjJifcL examined current vs. percei ved-ideal prpcedure,s for ♦ . 
determining Wogranf gya lyation btudjgets. ' Tfe /third ami final section • 
of the *ifepe if offers' three' concrete >Vcommerid^tTons for future actidn. The 
.^eW^rahodie/ in tt^is article are %ievresul t--of an extensive three yea£. 
effroft jfertT the* part of the task ^fo^e^embetshtp; * ■ , • * , - " 

~*\ ' *' ..„'":.'- ° ' ^HY"* EVALUATE? • 

Need tor Commitment to Program Evaluation ~ . ' \ . M . *\ . / 1 ^ • 

In an eYa characterized by- increasing' demands for accountabil ity ^and * , 
dy/fpdTina resources*,- program 4 evaluation has emerged as central to the 
provision. of .quality educational programs, and' requisite for compliance with' 

state and federal mandates'. - ' ' s H . 

** • / . *• 

*** " 

In light of mounting public pressure 'and recent legislative action, LEAs 
are being forced t& scrutinize, even more closely the use. of shrinking* 
education dollars. Apart from the issue of 'compliance,, many districts ■ 
are, beginning to .realize that program evaluation is essential" to *the t, 
effective u§e of system resources. It/is becoming increasingly important 
for educators to determine. what* works and what does not in order'to 'expend 
finite public resources, in. the most effective arid efficient manner.', 
However, few school districts' have adequate research and evaluation components. 
The' majority* of. LEAs throughout .the country function, for the most part, 
without the benqfit'.of these services- or-are forced to contract with outside 
"consultants as considerable expense. Thus, the question of resource • . 
allocation for program evaluation ij a pervasive pne which- must be addressed 
at the state and fe.de'ral leve.l,as well .as, at^the local level. 

t • *' " 

rogram^evaluation results serve two purposes. First, they provide program . 

staff with information necessary for making program decisions'. Second,* 

they provide pol.icy-makers (administrators) with information necessary 

for making policy decisions.- The execution of effective programs, like the 

formation of effecti ve^poLicy, dependsluppn one's ability to assess system 

needs and. resources and to interpret both in light of information concerning, 

program worth. The information requirements for administrators parallel 

.those of program staff, that is they both must address the following questions. 



• 1. What are the needs? > . , • % 

• 2, How are programs being implemented? 

t 3/ How effective are the programs? 

Program evaluation,, as defined by the Tas,k Fprce, -embraces each of the 
above questions. First, need identification (I.e., program planning) is^ 
addressed through context and/or. input evaluation. Second, monitoring 
program operations (i.e., program implementation) is^addressed througtT 
process evaluation. And thirds "assessing program" outcomes (i.e. ,' program ; 
impact) is. aSdressed through product evaluation; Ha£h of £hese, thr^e 
phases' of--the -evaluation have^' profound effect upon J)oth prog^m. and fc ; 
policy decisions. . • ; * . * *' 

The commitment £0 /use. evaluation findings cafri x fch?refore be^seen at two 
levels: 1) the program- level rand 2) tlje pol tcy % 'leveL ' 'Rlconkgndations 
concerning resource* allocation for program evaluation must address these 
specific concerns v as \^ell as _the ^'general* question of producing usable 
evaluation findings.' ^ . - 

There i*s 'perhaps -no better expression of a commitment to' use of evaluation 
findings than the adoption of a forma It policy statement by local governing 
boards affirming the same* A straightforward pcrMcy statement of commit- s 
ment to use of evaluation findings would be hard for 3 local governihg . 
board to turn down. People want- to know what their tax dollars (or other > 
funds) are buying. There is v general /agreement that the purpose' of educational* 
institutions is t<3 provide .learning opportunities* for students, not to * 
perpetuate programs "or practices that do not produce results. 

HOW- MUCH EVALUATION ft ENOUGH? ' \ 

Current vs. Perceived-Ideal Procedures for Determining Evaluation Budgets 

t * * 

The average family spends about as much per year to assess the merit of. its 
children's educational programs -as it does' to buy school lunches for the 
same children for pne week. This conclusion is based on a 1979' Division H 
questionnaire survey of 55 local-education-agency (LEAs) and ,14* state- 
education-agency (SEAs) evaluation directors ,(Note 1 ). This conclusion - 
accords, with results of two -other recent national surveys of evaluation •* ■ 
activities in U.S. education: one' by Lyon,- Doscher, McGranafcan, and 
Williams (Note 2) ;*the'vOthef by Webster and Stufflebeam. (Note 3).... x 

All three surveys suggest that dliring the late 1970's, U.S'. school districts 
with evaluation units allocated* only .3% of their total budgets for 
evaluation. Other major findings of the Division H-survey are listed below.* 

1. Both smaller (under 40,000 students) and larger school districts 'spend 
about $4.00 per pupil for evaluation- ($2.'00 locally funded) .but of a 
meaii per pupil expenditure of $1,500 for' the smaller LEAs and $1 ,70p 
for the Targer.LEAs. 0$ this $4.00, «$3.00 is spent on' outcome or 
process evaluation, $.6.0 .o'h progr^P lann1 '. n 9/ nee 4 s - as s es sni ent ' and " ' 
$.40 on dissemination/utilizatjo'n. • • ' ' • - * ' • » ~ 



2. When educational programs have, an N evaluation component, about, 1^% 
of the total program budget is ordinarily allocated for evaluation* 

^ For some of the largest federally funded programs* ie.g. /Title I* '/ ' • - 
and special education) these evalyatiofi allocations average .573%* 
For federally funded (e.g., Title IV-C) and nonfederal programs 
emphasizing innovative approaches to education— where documentation 
. of program acceptance and effectiveness may be more strongly emphasize^ 
—the .percentages of.*prog>am costs allocated for evaluation seem to 
average 4.5%. • / * " 

v 

3. This survey ^ows that a nationwide samp Te\Qf heads of LEA^ evaluation 
units- recbmmend a range of percentages of program^ cos tsrfbY < v : « 
evaluation—from 4-8%, on the* average— that exceeds the currently 
available average percentage-of-program-costs allocations for evaluation. 

* « . » *i 

4/ A majority; of these LEA evaluation directors (as well as a majority 
of the SEfi evaluation directors represented) recommend a slicing scale , 
* approach ~for determining allocations of program costs for evaluation, 
% with- lower percentages as program costs exceed centain levels. A 
majority of the evaluation units represented currently use such a 
sliding scale approach, with' the evaluation budget determined by the 
jspope of evaluation work required. ♦ The current average range of 
^percentages of prcigram costs for evaluation when such sliding scales > 
ar.e used is only.. about 1.5-5/5%. 

5. Overall, about three^fourths of the LEA evaluation directors judged 
their-units 1 evaluation rejforts to be an important factor in subr, 
sequent pr&gram operation; 'and about half \judged these reports important 
to subsequent decisions concerning progranufunding. There were modest * 

- pojnt-bi serial correlations between percentage of total LEA budget 
allocated for evaluation and (a) influence of evaluation reports on , 
subsequent 'funding (r = ,J21, £= .07, 1-tailed) and (b) influence of . 
evaluation fepbrts.on subsequent program operation (r ?■ .24, P = .04, *\ 
1-tailed).- Thus there is some- reason to believe that LEA level of 
effort for evaluation is related to the usefulnessyof evaluation ^ 
activities in decisions concerning prpgram funding atfd program revis/jfrn. 

6. Comments of these LEA and, SEA evaluators suggest possible reasons why 
evaluation does not have-a^higher importance in prpgram funding ajid 
operation;' or, more generally, why evaluation does not have a higher*' 
priority among LEA*, SEA, and federal educational leadership.). According 
to these , respondents , program evaluation is often seen as*tcfo costly;- 
educational leaders .Have 1 ittle training, or experience in using. 

. evaluation results; some evaluation's-, lack usefulness because they are 
. inconclusive or carried out independently from program development; 
and some LEAs^apparently do not have enough staff qualified. ,to conduct ■ 
high quality evaluatiwS. ' v . - * . * v t ' - % 

Since the respondents' to the Division H survey questiofw£i re .represents 
35% return 'rate: for LEA- evaluatbrs and a 25%*ret^rn rate foVSEA evaluators,; 

.caution must be used in generalizing .Uese survey results to the entire. 
Intended. sample; .or to all IL£ LEAs and SEAs. Jhe 'respondent -group does,**' 
however, represent a broad geographic distribution, apcC a large range of ^ 
L.EA ajid state\tudenVpopulation sizes. Also, the LZKs represented intliis 
survey,, taken together, acc'otmted for.near-ly 7% of the 1977 r 28 U.S. school- 

•age population. . ♦ ' '' >: 

- • -3- ' - - , . - . . • ' 



s One implication *of the Division H survey appears clear, educational 
evaluators need to increase their efforts to communicating evaluation 
results and, encouraging educational leaders to use these results. 
•Apparently, of every $10' allocated for analysis of data, only .about 
•$i: is spent on efforts to disseminate research results and encourage 
their use. 

Raizen and Ros&i (Note .4) 'paint, a simular pict«re ; at the Federal level.. 
According to thejr study, evaluation receives only S% of approximately 

* ~$1.4 billion in federal aid to education. ? ,They point out that a report 
by' the .National Research, Council suggested that Congress must be more 
specific about what it wants from evaluations of educatidna-1 programs. , 
And finally, they conclude that result's of evaluation studies conducted 
for the Education Department generally do .not get put to the field, 
and'proceed to suggest that at a minimum evaluation findings must be 

. communicated- to the primary 'audience. 



WHERE DO WE 60 FROM HERE? 



Three Recommendations 



The Division H Task Force on Resource Allocation has developed three 
recommendations concerning the funding of program evaluation: activities. 
The first two recommendations ..address important preconditions which should 
be present prior to establishing an evaluation "budget, while the third 
recommendation focuses on setting appropriate funding levels for conducting 
•evaluations using a sliding scale. 



".NUMBER " Federal, State and Local Education Agencies Should adopt a common 
1 definition of program evaluation which is consistent with the 

Standards for. Eval uations of Educational Programs, Projects, and • 
Materials promulgated by The Joint Committee on Standards for. • t • \ 
• Educational Evaluation. McGraw-Hill Book Company, New York, 
\\ , . 1981. This definition whould embrace the full range of evaluation 

activity fronrprogram planing (.need-assessment) to process, or 
■ implementation evaluation through product or outcome evaluation 
. • > to dissemination. „ 

All too often those who commission evaluation studies define program % * 
evaluation in very narrow terms. In most cashes evaluation is used • , 
synonymously with testing oFassessment programs. This tendency riot only serves 
•. as a barrier to effective communication between the evaluator and- the sponsor, 
' but also tends. 'to distort the amount of- support (funds') needed to conduct t 
. program eval uation, activities. 

I* ' . Asses'smeht an-d testing programs do have, a role to play- in assisting educators 

• with'the eval uat,fpn~6f program 'outcomes, namely, by providing^ ready data_^ 
ba-se.-s.However, Ao .neglect other evaluation considerations such as program. 
• . development, monitoring: and dissemination .of resets seriifusly undermines • 
/the resource 'allocation. process^ , \. \ .■ v 




Therefore, the Ta$R Force recommends the adoption pf a common definition 
of program evaluation,; one which embraces the full range of evaluation * 

.concerns from lAannirig.to dissemination of results. . - ; # J 

* ^ * » • -.• 

-tfUMBER a Hidi vi finals or groups who contniss^ion evaVuatjon studies mus.t 

2 "clearly delineate the uses to t which evaluation finaings will < 

. be put,. as well as the anticipated level of dissemination. « 
These twp pieces of information are essential When calculating 
the levet of support required for any one program evaluation 
effort." Knowledge^ the purpo§e(s) of 'the evaluation and > 
its potential audience not only influences funding and resource ~ 
levels, but alsoHme^cts an .methodological questions.- 

Therefor, the "Task FQrce recommends that requests 'for .evaluations be 

accompanied , by a statement of the use tcwhich the evaluation tiata will . 

be put, as.well as the projected scope of dissemination efforts-. 
• / 

r a , 

NUMBER Based' upon Task Force survey data (Note 1), it appears that 'the 

3 > most reasonable approach to setting evaluation budgets involves^ 

* . the use of a sliding scale. For the typical program evaluation * 
-(i.e^,* one which .encompasses program p\ttnnirftj; process evaluation; 
product evaluation"; and' dissemination) the optional level of 
•s^Eport appears toJrange from 4 to 8 percent of the total program' 
budget. ' " , 

Ohr survey revealed that'-of every $10 allocated for evaluation, slightly 
ovkr $4 was spent on product or outcome evaluation activities; $3, on 
process" or implementation' data gathering; nearly $2, on program 1 planning 
and r\eeds assessment; and*$l, on dissemination and utilization.. These 
average proportions of furlds spent during each evaluatipn phase were 
similar for both large and" small LEAs. < 

However, variability among LEAs ill the proportion of evaluation -funds spent > 
on each phase was guite high, particularly for the product/outcome and 
process/implementatton phases. These results are not surprising given that 
'program evaluations differ greatly in purpose and interft.- About # half of 
ttffe LEAs reported spending 30-50% of evaluation funds''during the product/ 
outcome evaluation phase; and about half spent, 20-40% during process/ 
implementation evaluation: Only 6% of the LEAs reported spending more 
than 20% of evaluation funds in dissemination and use of" data. 

The variable 'nature of ,eval uatipn studies make It difficult to approach 
resource allocation as an exact science/ Several factors" interact with 
the overall purpose ..and intent- of an evaluation to makej)udgeting a tricky 
•business. Once again the task Force survey provides valuable insight 
into -the question of what factors lead -to a, higher or lower than average 
level .of support for evaluation activities.- Sixty-nine LEA' and SEA evaluation 
directors were asked for written responses to two questions: "What are some, 
of the factors ,-which would make a program ^valuation 'cost morgA tharirfrhfe 
normai range?" and "What are some of th£ faotors .^ich would make'Sn* 
evaluation cost less ,.-<f\ ?" , ' 



Measurement and i nstrumentati on ;f actors 'leading to higher than average 
costs were: (1) field, collection of interview and observational data, 
in addition to collection of paper-and-pencil data (22% of respondents); 
(2) developing aew or original tests or questionnaires {20%); (3) purchasing 
commercially available tests or questionnaires (9%); and (4) a large number . 
of different instruments and measures required by "the evaluation design (7%). 

Other evaluation features or requirements leading. to higher than average 
costs were large program .size or complexity (14%); complex evaluation 
design— e.g., control groups, evaluation audit, etc. — (10%); and heavy data 
analysis and reporting requirements (10%). 

\* Measurement and instrumentation factors leading to more economical evaluation 
included: (1) using already available data, and/or al ready purchased or 
developed tests and questionnaires (16%); (2) evaluators^and program staff 
share the work or cost of data collection— including use of diagnostic 
.tests usable for both program operation and evaluation (7%); and (3) a 
small number of different instruments and measures required (6%). 

Other evaluation features or requirements leading to lower costs were - 
light data analysis and reporting requirements (10%); small program size 
or complexity (6*).;* and a design that is elegantly simple, or evaluation 
procedures that are famil iar or well standardized, (6%). 
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